Theoretical prediction of topological insulator in ternary rare earth chalcogenides 
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A new class of three-dimensional topological insulator, ternary rare earth chalcogenides, is theo- 
retically investigated with ab initio calculations. Based on both bulk band structure analysis and 
the direct calculation of topological surface states, we demonstrate that LaBiTe3 is a topological 
insulator. La can be substituted by other rare earth elements, which provide candidates for novel 
topological states such as quantum anomalous Hall insulator, axionic insulator and topological 
Kondo insulator. Moreover, YBiTe3 and YSbTe3 are found to be normal insulators. They can be 
used as protecting barrier materials for both LaBiTe3 and Bi2Te3 families of topological insulators 
for their well matched lattice constants and chemical composition. 

PACS numbers: 71.20.-b,73.43.-f,73.20.-r 



Introduction- A topological insulator (TI) is a novel 
quantum state, which has attracted great attention in 
condensed- matter physics recently [l|-|3|. TIs in two or 
three dimensions have both insulating bulk energy gap 
and gapless edge or surface states on the sample's bound- 
ary. The surface states of three-dimension (3D) TIs con- 
sist of odd number of massless Dirac cones, which are 
protected by the time-reversal symmetry. TIs can be 
most generally defined as a new state of quantum mat- 
ter whose effective electromagnetic action is given by the 
topological term S e = (0/27r)(a/27r) / d 3 a;dtE • B, with 
a the fine-structure constant and the parameter 9 = 
or 7r for trivial insulator and TI, respectively Many 
novel properties have been proposed as a result of this 
topological response [U-IH, which are interesting in both 
fundamental physics and device applications. In proxim- 
ity with an ordinary superconductor, TI also provides a 
new candidate for topological quantum computation Q . 

The first TI was theoretically predicted and experi- 
mentally observed in the HgTe quantum well [8, 9]. This 
work discovered the basic mechanism of band inversion 
driven by spin-orbit coupling (SOC), which serves as 
a template for most TIs discovered later. Since then, 
several TIs in both 2D and 3D have been theoreti- 
cally proposed and experimentally realized, Bii-zShr 
alloy[IS[!l|l, the famil y of Bi 2 Se 3 , Bi 2 Te 3 and Sb 2 Te 3 [l2|- 



|14(, the family of TlBiTe 2 and TlBiSe 2 \mUM • Man Y 
other materials are theoretically predicted and still wait- 



ing for experimental verification 3- 24 1 . Of particular in- 
terest are the magnetic atom doped topological insulators 
for the effects of magnetic impurities and ferromagnetism 
on the topological surface states 2^-27|. In this letter, we 
report a new family of ternary chalcogenides compound 
LnBT 3 (Ln = rare earth elements, B = Bi, Sb, and T 
= Te, Se), which share the same crystal structure with 
Bi 2 Se 3 family. LnBT 3 can be regarded as Bi 2 Se 3 with 
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FIG. 1: (Color online) Crystal structures of LaBiTe3 with 
three primitive lattice vectors denoted as il;2;3 for (a) type-I 
and (b) type-II lattices. The quintuple layers, Te-La-Te'-Bi- 
Te for type-I lattice, and Te-La-Te'-La-Te and Te-Bi-Te"-Bi- 
Te for type-II lattice are indicated by rectangles. The c axis 
of equivalent hexagonal lattice is along the z direction, (c) 
The first Brillouin zone is shown for the rhombohedral lattice 
as well as the 2D projected surface. 



half of the Bi atoms replaced by Ln. Unlike Bi 2 Se 3 fam- 
ily, in these materials inversion symmetry is not always 
preserved. Since lanthanum(La) and Yttrium(Y) have 
no 4/ electrons, LaBiTe 3 , YBiTe 3 , LaSbTe 3 and YSbTe 3 
are four simplest ternary rare earth chalcogenides which 
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have been synthesized in experiments 28]. In this work 
we focus on these four compounds based on ab initio cal- 
culations, among which we find that LaBiTe3 is a strong 
topological insulator. Moreover, La and Y can be re- 
placed by other rare earth elements which have magnetic 
moments due to the localized / electrons. These systems 
could provide realizations of the topological Mott insula- 
tor state [1^]. The versatility of rare earth elements will 
bring us new possibilities beyond Bi2Te3 family. 

Method and crystal structure- The electronic structures 
of LaBiTe 3 , YBiTe 3 , LaSbTe 3 and YSbTe 3 are calcu- 
lated in the framework of the density functional the- 
ory with Perdew-burke-Ernzerbof type generalized gradi- 
ent approximationfGGA) pjoj] . For our calculations, both 
the Vienna ab initio simulation pacfcaqefVASP')[3ll. |32 | 
with the projected augmented wave method [33j] and the 
BSTATE package 34j with plane-wave pseudo-potential 
method are employed. We adopt the 12x12x12 k-point 
grid for self-consistent calculations. The kinetic energy 
cutoff for the plane wave basis in BSTATE package is 
fixed to 340 eV. SOC is included in all of our calcula- 
tion, except in the calculations for the ionic optimization 
which is confirmed to be insensitive to SOC. 

LaBiTe 3 , YBiTe 3 , LaSbTe 3 and YSbTe 3 share the 
same rhombohedral crystal structure with the space 
group D 5 3 d(R3m) (see Refl28l and references therein). 
Here we take LaBiTe 3 as an example. Similar to Bi2Te 3 , 
the crystal consists of triangular layers stacking along 
[111] direction with A-B-C- • • order. Five atomic lay- 
ers form a quintuple layer (QL) with the order of Te- 
X-Te-X'-Te, where X and X' denote La or Bi organized 
in certain pattern, as discussed below. Though the lat- 
tice parameters have been measured by experiments, up 
to our knowledge, the relative position between M and 
M' has not been fully resolved experimentally. Thus we 
compare the total energy obtained in ab initio calcula- 
tion for different configurations to search for the most 
stable structure. Our result indicates that the structures 
with La and Bi separated in different atomic layers are 
more stable than those with La and Bi mixed in the same 
atomic layer. There are two structures which have lowest 
energy. Because the energy difference between the two 
configurations are very small, in the following we will 
study both of them separately, (i) Type-I structure con- 
sists of stacking quintuple layers along [111] direction, 
each of which is ordered as Te-La-Te-Bi-Te, as shown 
in Fig. 1(a). This structure is inversion asymmetric, 
(ii) Type-II structure consists of stacking pairs of quin- 
tuple layers along [111] direction, ordered as Te-La-Te'- 
La-Te and Te-Bi-Te"-Bi-Te. This structure is inversion 
symmetric. The experimental lattice constants and opti- 
mized atomic coordinates for the two structures are listed 
in Table I. 

Type-I structure. We start by computing the band 
structures for LaBiTe 3 , YBiTe 3 , LaSbTe 3 and YSbTe 3 
in the type-I crystal structure. As examples, Fig. 2 shows 
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FIG. 2: Band structures for(a) LaBiTe 3 and (b) YBiTe 3 of 
type-I structures and (c) LaBiTe3 and (d) YBiTe3 of type- 
II structures. The strengths of spin-orbit coupling(SOC) are 
tuned to be 50% (dashed purple lines), 89% (thin red lines) 
and 100% (thick black lines) respectively, in (a) and (b). The 
Fermi energy is set to zero. The energy gap decreases to 
zero and re-opens as SOC increases from zero to 100% for 
LaBiTe3, showing a clear band anti-crossing feature. The 
Dirac type level crossing at 89% SOC is highlighted in the 
inset, (e) Brillouin zone for this class of materials. The 
four inequivalent time-reversal-invariant points are F(0;0;0), 
L(n; 0; 0), F(ir; ir; 0) and T(-7r; 7r; 7r). The surface Brillioun 
zone is projected from the 3D bulk Brillouin zone into the 
plane of the atomic layer. 



(a) LaBiTe 3 



(b) YBiTe 3 




FIG. 3: The band dispersion for an semi-infinite surface of 
(a) LaBiTe 3 , (b) YBiTe 3 , (c) LaSbTes and YSbTe 3 of type-I 
structures. The inset of (a) highlights the surface states near 
the Dirac cone. 
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TABLE I: The experimental lattice parameters [2^] a and c of hexagonal lattice in unit of angstrom, the theoretically optimized 
atomic positions and band gaps. Type-II structure has twice the length of type-I along c axis (see Fig.l) and thereafter has 
double atoms. The atomic coordinate is (0,0, zqc) in the rhombohedral primitive unitcell. We listed zq values for all atoms. 
These Te atoms as inversion centers of type-II are marked as Te' and Te". The energy gap(_E s ) is in unit of eV. 







Type-I 






Type-II 






LaBiTe 3 


YBiTe 3 


LaSbTe 3 


YSbTe 3 


LaBiTe 3 


YBiTe 3 


LaSbTe 3 


YSbTe 3 


a 


4.39 


4.46 


4.24 


4.47 


4.39 


4.46 


4.24 


4.47 


c 


30.20 


31.65 


30.40 


30.32 


60.40 


63.30 


60.80 


60.64 


Te(Te'/Te") 


0.0 


0.0 


0.0 


0.0 


0.0/0.5 


0.0/0.5 


0.0/0.5 


0.0/0.5 


Te 


0.20876 


0.22243 


0.20426 


0.21850 


± 0.10604 


± 0.10764 


± 0.10702 


± 0.10839 


Te 


-0.20893 


-0.21228 


-0.21136 


-0.21306 


± 0.39572 


± 0.38883 


± 0.39790 


± 0.39076 


Bi(Sb) 


0.40274 


0.40161 


0.40102 


0.40156 


± 0.19922 


± 0.19873 


± 0.19846 


± 0.19801 


La(Y) 


-0.39938 


-0.39416 


-0.40111 


-0.39582 


± 0.29951 


± 0.30219 


± 0.29913 


± 0.30120 


E g 


0.12 


0.12 


0.15 


0.28 


0.07 


0.07 
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the results for LaBiTe3 and YBiTe3. Both of them have 
similar band gap 0.12 eV around the T point. The 
band gap of YBiTe3 is close to the previous experimen- 
tal value f35l| 0.18 ~ 0.23 eV. As mentioned above, due 
to the lack of inversion symmetry, the parity criteria 
for topological insulators [10j does not apply. Alterna- 
tively, we study the evolution of conduction and valence 
bands by tuning the SOC strength artificially. Without 
SOC, any insulator must be topologically trivial, so that 
a topological insulator phase can be identified by study- 
ing the topological phase transition at some critical SOC 
strength|8]. A Dirac type level crossing between valence 
and conduction bands se par ate the topologically trivial 
and nontrivial phases. |36| The continuous tuning of 
SOC is obtained by changing the speed of light c in the 
VASP package. As shown in Fig. 2(a), upon the increase 
of SOC strength, the band gap of LaBiTe3 decreases 
and vanishes on a ring in Brillouin zone (BZ) near T at 
89% SOC and opens again, which indicates a topological 
phase transition 37] and suggests that LaBiTe3 is a topo- 
logically non-trivial insulator with full SOC strength. In 
contrast, the band gap remains finite upon turning on 
SOC for YBiTe3, which is thus a topologically trivial in- 
sulator. We perform the same calculations for LaSbTe3 
and YSbTe3, and find both of them are topologically triv- 
ial insulators. 

The existence of topological surface states is one of the 
most important physical consequence of the TIs. To fur- 
ther confirm the topological nature of the proposed mate- 
rials and study the detailed features of the surface states, 
we obtain the dispersion of surface states based on ab 
initio method. We construct Tight-binding(TB) Hamil- 
tonian of the semi-infinite system with Maximally local- 
ized Wannier functions(MLWF)[H,[39i basis for LaBiTe 3 
family materials, and then apply the standard Green 
function iteration method 0, 4l[ to obtain the surface 
density of states. The detail of this method can be found 
111 RefS In Fig. 3 we can see clearly that the topo- 
logical surface states form a single-Dirac-cone at the F 
point in 2D BZ within the bulk gap for LaBiTe3, while 



no such surface states are observed for YBiTe3, LaSbTc3 
and YSbTe3. This result determines the topologically 
non-trivial nature of LaBiTe3 without any doubt. From 
this calculation, we can further obtain the Fermi veloc- 
ity for LaBiTe3 to be 4.0 xl0 5 m/s, similar to that of 
Bi 2 Se 3 [I3. 

Type-II structure. The band structures for LaBiTe3 
and YBiTe3 of type-II are shown in Figs 2(c) and 2(d). 
Their band gaps are 0.07 eV near the T point, smaller 
than those of type-I structures. Due to inversion symme- 
try, we can follow the parity criteria proposed by Fu and 
Kanefioj] to calculate Z2 topological index. The product 
of the parities of the Bloch wave functions are determined 
for the occupied bands at all time-reversal-invariant 
momenta T(0, 0, 0), L(tt, 0, 0), F(w, tt, 0), T(tt, tt, tt) in BZ. 
For LaBiTe3, we find that the product of parities at T is 
" + " and those at L, F, T are all " - " . Thus LaBiTe 3 is 
topological nontrivial with Z2 index (1;000). For YBiTc3 
and YSbTe3, we find that the products of parities at 
r, L, F, T are all " - " , so that both of them are topologi- 
cally trivial with Z2 index (0;000). In addition, LaSbTe3 
is found to be a semimetal. For both type-I and -II struc- 
tures, it is noted that the states near Fermi energy are 
partially composed by La/Y— 5d z 2 as well as Bi/Sb-6p z 
and Te-5p z orbitals. 

Discussion- In summary, our ab initio calculation has 
verified that LaBiTe3 is a strong topological insula- 
tor. The substitution of La in LaBiTe3 with other rare 
earth elements will open a wide platform to study var- 
ious topological non-trival behaviors. It is found that 
many isostructural stoichiometric compounds similar to 
LaBiTe3 also exist in the family of ternary rare earth 
chalcogenides (jjjl . such as LnBiTe3 (with Ln standing 

for La,Ce,Pr, ,Lu or Y) and LnSbTe3, many of which 

have intrinsic magnetic moments due to the localized / 
electrons (e.g. Pr, Sm or Gd). Since the strong corre- 
lation effect induced by / electrons can not be properly 
taken into account in the current GGA calculations, we 
will leave the detailed study of these materials for the 
future work but discuss the possible states induced by 
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strong electron correlation. First, if the coupling between 
magnetic moments is ferromagnetic, quantum anomalous 
Hall effect may emerge in thin films or heterostructures 
of LnBiTe 3 when the band structure is normal for one 
spin while inverted for the other, similar to the propos- 
als in Hgx_ x Mn g Te[Iil and Bi?_ Je-,Se^(27t. Second, if 
the magnetic coupling is ferromagnetic within one atomic 
layer but anti-ferromagnetic between two neighboring 
atomic layers, the dynamical axion field may be realized, 
which is proposed as a novel optical modulators deviceta . 
The recently proposed topological Kondo insulator [43j 
may also exist in this class of materials due to the ex- 
istence of / electrons. Besides the topological insulators 
the newly proposed materials have another important po- 
tential application. As we have shown, some materials 
in this family are trivial insulators such as YBiTe3 and 
YSbTe3. Due to their similarity with the Bi2Te3 family, 
the trivial insulators in this new family are natural candi- 
dates of barrier materials for thin films or quantum wells 
of the TIs in the Bi2Te3 family and the new LaBiTe3 fam- 
ily. Especially, YBiTe 3 (a=4.46 A) - LaBiTe 3 (a=4.39 
A) and YBiTe 3 - Bi 2 Te 3 (a=4.3835 A)[H] pairs have the 
best-matched lattice constants. Such a barrier material 
may help protecting a clean surface of TI and greatly 
improve the surface condition in transport experiments, 
which is a main difficulty in the current experiments. 
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